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Abstract 

The  increasingly  complex  battlefield  environment  drives  the  requirement  for  the  presentation  and 
interactive  control  of  the  endless  stream  of  information  arriving  from  a  diverse  collection  of 
sensors  deployed  on  a  variety  of  platforms.  At  best,  the  situational  awareness  picture  is 
fragmented  without  the  benefit  of  data  fusion  and  correlation  to  present  a  true  picture  of  the 
battlespace  from  all  information  sources.  Collaboration  and  interaction  is  also  needed  for 
operators  within  a  control  center  and  among  remote  geographic  locations.  The  need  to  display 
and  manipulate  real-time  multimedia  data  in  a  battlefield  operations  control  center  is  critical  to  the 
Joint  Commander  directing  air,  land,  naval  and  space  assets.  The  Interactive  DataWall  being 
developed  by  the  Advanced  Displays  and  Intelligent  Interfaces  (ADII)  technology  team  of  the 
Information  Directorate  of  the  Air  Force  Research  Laboratory  (AFRL/IF)  in  Rome,  New  York  is 
a  strong  contender  for  solving  the  information  management  problems  facing  the  21st  century 
military  commander.  It  provides  an  ultra  high-resolution  large  screen  display  with  wireless 
interaction.  Commercial  off-the-shelf  technology  has  been  combined  with  specialized  hardware 
and  software  developed  in-house  to  provide  a  unique  capability  for  multimedia  data  display  and 
control. 

1 .  Introduction 

The  problem  is  the  inability  to  effectively  display  and  manipulate  large  amounts  of  real-time, 
multimedia  data  in  a  Command  and  Control  (C2)  environment.  Migration  to  electronic  media  for 
mission  planning  is  progressing.  However,  conventional  media  such  as  large  paper  maps  with 
acetate  overlays  and  Plexiglas  boards  for  grease  pencil  annotation  are  still  the  norm.  The 
transition  has  been  delayed  since  new  methods  of  operation  are  often  slow  to  be  accepted,  the  use 
of  individual  workstations  diminishes  the  big  picture  in  a  C2  environment,  and  an  intuitive, 
unencumbered  means  of  Human  Computer  Interaction  (HCI)  is  limited. 
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The  approach  is  to  utilize  commercial  of  the  shelf  (COTS)  products  to  the  greatest  extent  to 
create  a  very  high-resolution,  tiled  wall  display.  It  is  usable  by  both  operators  who  are  in  close 
proximity  to  the  screen,  and  by  commanders  who  typically  work  at  a  distance.  Multiple  users  can 
directly  manipulate  the  wall  display  using  speech  recognition  and  wireless  pointing  devices  for 
unencumbered  interaction. 

The  uniqueness  of  the  AFRL/IF  Interactive  DataWall  is  twofold.  Its  enhanced  computer  display 
capability  tiles  the  output  of  multiple  computer  displays  into  a  single  workspace.  The  current 
configuration  has  a  total  display  resolution  of  approximately  5.8  million  pixels  over  a  12’  x  3’ 
screen  area.  Its  enhanced  HCI  capability  allows  for  wireless  interaction  with  the  display.  Speaker 
independent  speech  recognition  via  wireless  microphones  and  conventional  mouse  functionality 
via  camera-tracked  laser  pointers  have  been  incorporated.  Further,  the  laser  pointer’s  function  has 
been  enhanced  to  provide  an  electronic  grease  pencil  capability. 

2.  High  Resolution  Tiled  Display 

A  tiled  display  consists  of  NxM  distinct  display  devices,  such  as  video  projectors,  each  displaying 
a  portion  of  an  entire  screen  area.  The  Interactive  DataWall’ s  display  consists  of  three 
horizontally  tiled  video  projectors  producing  a  very  high-resolution,  near  seamless,  large  screen 
display  (Fig.  1).  Properly  balanced  color  and  brightness,  and  proper  alignment  of  the  display 
devices  is  critical  to  minimize  any  distractions  caused  by  the  seams  between  image  tiles. 
Variations  in  chromaticity  and  luminosity  among  tiles  can  cause  inconsistency  in  color  and 
brightness  across  the  screen.  Vertical  or  horizontal  disparity  of  the  tiled  images  can  cause 
segmentation  of  objects  at  the  seams.  Gaps  between  image  tiles  can  cause  discontinuity  of  the 
imagery. 


Figure  la.  AFRL/IF’ s  Interactive  DataWall 


3/4  View 


Front  View 


Figure  lb.  1x3  Video  Projectors  Configuration 

The  rationale  for  this  type  of  display  is  fourfold.  First,  a  large  screen  display  provides  a  global 
view  of  the  information  space,  and  allows  the  users  to  make  comparisons  and  find  relationships 
between  items.  It  also  makes  collaboration  within  a  localized  working  environment  much  more 
effective.  It  provides  multiple  users  with  a  common  display  medium,  coordinating  data  from 
multiple  workstations  and  monitors.  It  provides  a  large  canvas  to  present  multiple  windows  with  a 
variety  of  information.  Windows  can  be  spread  out  instead  of  continuously  bringing  the  active 
window  to  the  foreground.  Windows  can  be  maximized  to  the  total  12’  x  3’  screen  area. 

Second,  high-resolution  is  desired  because  AFRL/IF  is  researching  the  Interactive  DataWall  for 
military  Command  and  Control,  which  requires  the  display  of  various  kinds  of  data.  This  data  can 
include  terrain  models  overlaid  with  computer-generated  imagery,  digital  maps,  textual 
information,  as  well  as  live  and  recorded  video.  Although  a  conventional  projection  system  allows 
a  wide  range  of  image  sizes,  and  could  provide  the  necessary  large  screen  display  capability,  the 
display  quality  will  inevitably  decrease  as  the  display  area  increases  and  pixels  become  larger.  The 
DataWall  is  intended  to  be  used  both  at  a  distance  and  at  close  range.  A  large  screen  display's 
utility  is  significantly  reduced  if  imagery  loses  important  detail  or  text  becomes  difficult  to  read  at 
close  proximity.  Even  today,  large  paper  maps  with  acetate  overlays  are  used  for  mission  planning 
in  the  command  centers.  Therefore,  near  paper  map  quality  is  required  to  effectively  replace 
conventional  data  sources  with  electronic  media. 

Third,  tiling  several  images  either  horizontally  and/or  vertically  provides  a  wider  field  of  view,  and 
allows  displays  of  unlimited  aspect  ratios  (i.e.  the  height  to  width  ratio)  to  be  created.  Simply 
increasing  a  projector's  display  area  will  introduce  pixelation  problems,  and  will  decrease  the 
brightness  of  the  image.  In  a  tiled  configuration  each  projector  is  displaying  a  small  portion  of  the 
entire  display  area.  The  combined  image  is  much  brighter  than  a  single  projector  displaying  an 
image  across  the  same  screen  area.  By  limiting  each  projector's  display  area,  the  projected  light  is 
more  concentrated.  Direct  view  displays  are  much  brighter  and  less  susceptible  to  ambient  light 
problems  than  projection  systems.  However,  implementing  a  single  element  direct  view  display 
creates  a  myriad  of  new  problems.  Scaling  already  bulky  CRTs  or  flat  panels  is  neither  practical 


nor  economical.  Scaling  up  CRTs  beyond  a  40-in  diagonal  would  require  glass  envelopes  that  are 
heavy  and  high  voltages  that  are  radiation  hazards.  Fabrication  of  very  large-scale  flat  panels 
would  require  the  development  of  very  expensive  equipment  [Alphonse  and  Lubin,  1992].  Larger 
flat  panel  displays  are  becoming  available,  but  are  far  from  the  resolution  capacity  of  the  CRT 
systems.  In  addition,  the  state-of-the-art  in  display  technology  is  limited  to  devices  that  are  only 
capable  of  resolutions  on  the  order  of  2500  x  2000  pixels.  Tiling  the  highest  resolution  display 
devices  in  an  NxM  configuration  increases  the  display  resolution  NxM-fold. 

Last,  projectors  were  utilized  to  minimize  the  seams  between  the  image  tiles.  A  common 
implementation  for  tiled  displays  is  to  use  direct  view  monitors.  All  currently  available  direct  view 
monitors  have  frames  that  enclose  a  certain  amount  of  necessary  electronics  surrounding  the 
display  screen.  Therefore,  there  is  no  way  to  effectively  abut  direct  view  display  elements  without 
having  a  gap  between  them. 

2.1  Optimizing  a  Tiled  Image 

A  significant  challenge  of  producing  a  seamlessly  tiled  display  is  ensuring  the  projectors  are 
properly  aligned,  with  correctly  balanced  color  and  brightness  across  the  individual  projector 
images.  The  image  should  resemble  a  monolithic  display.  Varying  colors  and  brightness  caused  by 
chromaticity  and  luminosity  variations  will  give  the  impression  of  a  mosaic  rather  than  a  single 
continuous  image.  There  should  be  no  segmented  lines  or  gaps  between  the  tiles  caused  by 
inaccurate  alignment  and  geometric  display  distortions. 

The  ADII  team  developed  an  X  windows-based  application  that  generates  several  interactive  test 
patterns  [Jedrysik,  et  al.,  1998],  It  provides  a  comprehensive  testing  environment  for  a  seamlessly 
tiled  NxM  display.  This  application  greatly  improves  composite  display  image  quality  by  providing 
a  more  accurate  and  less  time  consuming  method  for  registration,  color  balance  and  alignment  of 
the  video  projectors.  The  patterns  were  designed  to  accomplish  the  following: 

•  Ensure  each  projector  is  displaying  all  pixels  around  its  border 

•  Allow  horizontal  alignment  of  side  by  side  tiled  displays  and  include  a  capability  for 
vertical  alignment  in  future  tiled  displays  greater  than  lxN 

•  Ensure  accurate  color  balance  among  all  projectors  for  continuous  color  balance 
across  the  display 

•  Minimize  geometric  display  distortions  such  as  pincushioning,  keystoning,  and 
vertical/horizontal  nonlinearity  (Fig.  2) 

•  Test  the  capacity  of  the  display  system  to  resolve  very  high-resolution  images 


Correct 


Pincushioning 


Horizontal 


Nonlinearity 


Figure  2.  Geometric  Display  Distortions 

All  the  video  projectors  used  in  the  DataWall  implementations  at  the  ADII  Visualization 
Laboratory  are  either  Cathode  Ray  Tube  (CRT)  based  systems  composed  of  separate  red,  green, 
and  blue  CRTs,  or  Liquid  Crystal  Display  (LCD)  based  systems.  A  key  advantage  of  the  CRT 
based  display  systems  is  the  ability  to  correct  display  image  geometry.  Display  distortions  can  be 
isolated  to  specific  components  and  adjusted  in  a  number  of  ways  both  mechanically  and 
electronically.  Individual  CRTs  and  isolated  portions  of  the  display  image  can  be  adjusted 
independently.  However,  a  significant  amount  of  effort  is  required  to  optimize  the  display  image. 
Most  LCD  projectors  have  the  distinct  advantages  of  being  much  brighter,  smaller,  lighter,  and 
easier  to  set-up.  Their  shortfalls,  however,  include  lower  resolution  and  the  inability  to  correct 
display  distortions  through  geometry  adjustments.  The  latter  necessitates  precision  optics  to 
minimize  these  distortions  in  a  tiled  configuration. 

All  CRT  video  projectors  have  some  type  of  internal  test  pattern  generator  to  aid  in  focusing  and 
converging  the  CRTs.  Each  provides  a  step-by-step  process  that  includes  manual  focus  and 
electronic  adjustment  of  the  CRTs  to  correct  geometric  display  distortions.  Although  quite 
helpful,  the  patterns  available  are  limited.  None  of  the  LCD  projectors  in  the  ADII  Lab  provide 
any  kind  of  internal  test  pattern  generation.  Although  the  CRT  projectors’  built-in  patterns  can  be 
used  for  some  of  the  preliminary  tuning,  they  were  never  intended  to  address  the  many  issues 
associated  with  the  DataWall’ s  very  unique  multiple  projector  display  configuration.  Tiled 
displays  have  very  specific  needs  with  regard  to  alignment  and  color  balance.  A  conventional  large 
screen  display  would  typically  only  consist  of  a  single  projector,  where  exact  image  positioning  on 
the  projection  screen  and  color  balance  with  other  projectors  aren't  required.  Alignment  and  color 
balance  are  critical  for  tiled  configurations  to  reduce  the  distraction  of  the  seams  and  to  provide  a 
continuous  display  image.  Three  examples  of  the  patterns  developed  by  the  ADII  team  are 
depicted  below  in  1x3  configurations.  They  are  used  for  aiding  alignment,  color  balance  and 
focus,  respectively  (Fig.  3). 


Figure  3.  Three  sets  of  test  pattern  examples  represented  in  1x3  tiled  configurations 


An  important  aspect  of  the  ADII  test  pattern  environment  is  that  rather  than  using  any  internal 
test  pattern  capability,  the  projectors  are  being  driven  by  the  computer  system.  This  system  also 
generates  the  display  imagery  in  the  working  environments.  This  provides  a  more  accurate 
representation  of  a  working  environment  display,  and  allows  for  a  more  accurate  test  of  the  entire 
display  system.  All  external  and  internal  distortions  are  taken  into  account.  These  include 
problems  introduced  by  the  computer,  communication  cabling,  distribution  hardware,  and  the 
projectors.  This  ensures  more  accurate  tuning  of  the  displays  to  produce  better  and  more 
consistent  images  than  would  otherwise  be  provided  using  internally  generated  patterns. 

2.2  Window  Management 

The  Interactive  DataWall  uses  X-META-X,  a  COTS  software  package  from  X-Software.  X- 
META-X  provides  a  meta  window  manager  functionality,  merging  the  separate  X  Window 
managers  of  the  individual  display  drivers.  A  conventional  X  Windows-based  tiled  display  can  run 


on  either  a  single  Unix  workstation,  with  multiple  graphics  cards  (multi-headed),  or  multiple 
networked  Unix  workstations.  Each  graphics  card  in  a  multi-headed  workstation  runs  a  separate 
X  Window  manager.  Cursor  movement  across  the  individual  display  tiles  is  permitted,  but 
windows  can  neither  be  repositioned  nor  resized  beyond  the  boundaries  of  each  display  tile.  The 
same  windowing  limitations  apply  to  tiled  displays  running  on  multiple  workstations. 

X-META-X  provides  two  significant  features.  It  allows  windows  to  be  repositioned  anywhere  on 
the  tiled  display  and  lets  them  be  resized  to  the  maximum  size  of  the  total  display  area.  X-META- 
X  is  neither  a  stand-alone  X  Window  server  nor  a  specialized  window  manager.  The  X-META-X 
proxy  is  stacked  between  the  X  Window  server  of  the  workstation  and  the  X  Window  clients  via 
inter-process  communication.  It  is  executed  instead  of  the  X  Window  server,  which  in  turn  calls 
the  server  itself.  Managing  the  communication  between  servers  and  clients,  X-META-X  breaks 
the  fixed  relation  between  display  area  and  communication  port  of  one  X  server  or  multiple  X 
servers  simultaneously. 

The  total  composite  display  resolution  is  not  limited  by  X-META-X,  but  by  the  X  Windows 
protocol  which  codes  coordinates  as  16  bit  signed  integers.  Thus,  the  maximum  meta  display 
resolution  allowed  is  32,767  x  32,767  pixels. 

3.  Wireless  Interaction 

The  purpose  of  the  Interactive  DataWall  is  much  more  than  providing  a  high-resolution  summary 
device.  It  embraces  the  notion  of  a  truly  interactive  environment  that  avoids  the  tethers  of 
conventional  human-computer  interfaces  such  as  keyboards  and  mice.  The  Interactive  DataWall  is 
an  environment  that  allows  unencumbered  user  interaction  from  various  locations  near  and  far. 
Speech  recognition  via  wireless  microphones  and  conventional  mouse  functionality  via  camera- 
tracked  laser  pointers  provide  wireless  interaction. 

3.1  Camera  Tracked  Laser  Pointer 

To  provide  a  wireless  mouse-like  mode 
of  input,  a  camera-tracked  red  laser 
pointer  is  used.  Three  video  cameras 
equipped  with  red  filters  are  positioned 
behind  the  screen  atop  each  video 
projector  (Fig.  4).  The  cameras’  views  of 
the  screen  are  dark  fields  until  the  laser 
dot  comes  into  a  camera’s  field  of  view. 

The  live  video  from  the  three  cameras  is 
processed  and  the  data  is  subsequently 
sent  to  the  display  driver  for  proper 

positioning  of  the  cursor.  It  allows  all  the  Figure  4.  Video  Cameras  Mounted  Atop  Video  Projectors 
functionality  of  a  conventional  mouse 

including:  dragging/dropping  windows,  resizing  windows,  and  interacting  with  graphical  user 
interface  (GUI)  widgets  such  as  buttons  and  scroll  bars.  An  important  aspect  of  the  laser  pointer 


input  device  is  that  it  is  essentially  application  independent.  Since  the  Interactive  DataWall  runs  on 
a  Unix  platform  under  X  Windows,  the  application  must  be  X  Windows  compliant  however.  The 
only  other  requirement  is  it  must  have  a  GUI. 

3.1.1  PC/Video  Capture  Card  Implementation 

The  original  laser  pointer  tracking  implementation  consists  of  three  Personal  Computers  (PCs) 
equipped  with  video  capture  cards.  These  computers  provide  a  frame  by  frame  screen  capture  for 
each  video  camera  positioned  behind  the  screen.  The  frames  are  analyzed  via  ADII  developed 
software  on  the  PCs  and  subsequently  transmitted  to  the  display  driver  for  cursor  positioning. 

Although  quite  effective,  the  process  suffers  several  limitations.  First,  the  system  cannot  operate 
in  real  time.  The  approach  requires  that  the  camera  and  computer  complete  a  frame  before 
analysis  can  begin.  The  delay  penalty,  therefore,  is  never  less  than  the  time  required  to  complete  a 
given  frame. 

Second,  both  resolution  and  update  rate  are  constrained  by  the  processing  power  available.  A 
minimally  acceptable  320  x  240  pixel  image  at  15  frames/second  fully  consumes  the  resources  of  a 
current  high  end  PC. 

Lastly,  the  cost/benefit  ratio  may  be  difficult  to  justify  as  even  a  minimal  system  can  cost  several 
thousand  dollars,  and  multiple  PCs  are  required  in  these  multi-camera  implementations  (i.e.  one 
PC  per  video  camera/display  tile). 

3.1.2  Custom  Laser  Pointer  Tracking  Hardware 

To  address  the  limitations  of  the  PC/ Video  Capture  Card  implementation,  the  ADII  team  invented 
specialized  hardware  to  track  the  laser  pointer.  It  functions  in  near  real  time,  with  readily 
expandable  resolution,  and  at  a  small  fraction  of  the  cost. 

The  device  combines  a  microcontroller,  video  processing  logic,  a  pair  of  counters,  and  real  time 
control  logic,  which  together  track  the  pointer  image.  The  process  involves  synchronizing  the 
counters  to  follow  the  camera  video.  At  the  point  in  time  when  the  camera  “sees”  the  pointer,  the 
counters  will  contain  values  representative  of  the  pointer’s  position  on  the  display  surface.  These 
values  are  then  passed  on  to  the  display  driver  in  near  real  time  via  serial  cable  to  position  the 
cursor. 

In  order  to  evaluate  the  effectiveness  of  the  device,  a  side  by  side  comparison  between  it  and  a 
200MHz  Pentium  PC  implementation  was  conducted.  The  exercise  confirmed  several  significant 
advantages  of  the  new  system. 

First,  speed :  The  device  can  easily  produce  a  detection  each  time  the  video  scans  the  pointer  while 
the  PC  system  can  only  consistently  maintain  a  detection  rate  of  20-25%  of  the  scan. 


Second,  timeliness :  The  device  begins  to  report  a  detection  almost  immediately  after  the  video 
signal  “sees”  the  pointer  while  the  PC  system  must  always  complete  a  full  frame  before  its  analysis 
can  even  begin. 

Third,  resolution :  In  order  to  sustain  a  10-15Hz-update  rate,  the  PC  version  is  limited  to  a 
resolution  of  320  x  240  pixels.  The  device,  is  capable  of  512  x  480  pixels,  and  can  readily  be 
increased  to  any  practical  video  resolution  without  penalty. 

Fourth,  and  finally  cost :  The  original  device  was  assembled  at  a  cost  of  less  than  $150  while  the 
cost  of  the  PC  system  was  in  excess  of  $2500. 

Although  the  prototype  device  was  initially  used  to  track  the  position  of  a  laser  pointer  on  a  rear 
projection  video  display,  it  can  be  easily  adapted  to  scan  a  pointer  image  on  any  projection  (rear 
or  front)  or  non-projection  direct  view  display,  including  printed  material  such  as  a  paper  map. 
The  technique  also  lends  itself  well  to  multiple  panel  displays  as  both  the  cost  and  complexity  of 
the  device  scale  linearly. 

3.1.3  Laser  Pointer  Tracking  Software 

There  are  three  phases  involved  in  using  a  laser  pointer  with  the  Interactive  DataWall.  The  first 
phase  is  the  detection  of  the  laser  dot  on  the  display  surface,  the  second  is  the  registration  of  the 
dot  in  reference  to  the  displayed  image,  and  the  third  is  the  manipulation  of  the  systems  resources 
to  support  interactivity. 

3. 1.3.1  Detection 

The  first  version  of  the  detection  system  relies  on  a  PC  with  a  video  capture  card.  The  video 
capture  card  captures  a  single  frame  of  video  from  a  camera  behind  the  screen.  An  algorithm  then 
parses  the  information,  based  on  the  brightness  and  color,  determines  whether  an  instance  of  a 
laser  dot  is  present.  Upon  a  successful  detection,  the  program  sends  the  location  of  the  laser  dot 
over  the  Local  Area  Network  (LAN).  While  the  software  was  a  necessary  progression  by 
providing  quick  prototyping,  it  does  present  some  limitations.  The  video  card  was  only  able  to 
capture  each  frame  at  320  by  240  pixels  with  a  24-bit  color  depth  at  about  10-15  Hz. 

The  second  version  of  the  detection  system  uses  the  in-house  developed  custom  tracking 
hardware.  This  device  is  capable  of  detecting  a  spike  in  the  NTSC  signal  from  the  camera.  It 
informs  the  display  computer  of  the  laser  dot’s  location  through  a  serial  port  when  the  signal 
surpasses  a  customizable  threshold.  The  hardware  solution  provides  an  increased  resolution  and 
speed  by  being  able  to  analyze  512  by  480  pixels  at  60  Hz. 

3. 1 .3.2  Initialization 

It  is  difficult  to  correctly  register  the  laser  dot’s  location  in  reference  to  the  display  image.  Both 
detection  methods  have  a  lower  resolution  than  the  display  with  which  they  interface.  Two 


algorithms  were  developed  to  handle  the  interpolation  and  extrapolation.  Each  requires  the 
operator  to  use  the  initialization  software  prior  to  executing  the  runtime  system. 

■  The  first  algorithm  relies  on  a  simple  algebraic  equation.  The  user 

is  presented  with  a  black  screen  and  a  series  of  white  dots.  Each 
dot  must  then  be  illuminated  with  the  laser  pointer.  The  dots 
appear  one  at  a  time  and  create  the  stepped  pattern  as  seen  in 
Figure  5.  The  dashed  line  represents  the  active  field  that  the  each 
initialization  point  is  addressing.  Each  pair  of  dot  locations  are 
responsible  for  correcting  the  location  of  a  laser  instance  within  its 
bounds.  The  initialization  software  stores  the  sixteen  locations  per 
Figure  5. 16-Point  screen  into  a  file  and  exits.  This  algorithm  does  a  good  job  of 

correcting  within  the  zones,  but  can  cause  incorrect  interpolation 
when  traversing  between  the  edges  of  two  zones.  This  method  also  requires  the  runtime  support 
to  calculate  the  interpolation  and  extrapolation  at  runtime. 

The  second  algorithm  alleviates  the  problem  of  moving  between 
seems  and  reduces  the  number  of  initialization  points.  The  user  is 

presented  with  the  same  black  background,  but  illuminates  only  j||  hHH||4| 

four  initialization  points  (Fig.  6).  The  drastic  reduction  in  points  l|| , ;  ;  , ; 

does  not  reduce  the  accuracy  because  the  interpolated  and  l||  S9H§i:j!  | 

extrapolated  values  use  a  more  robust  rhomboidal  algorithm.  A  ||i  :  \  :  :  s9gn|||i 

file  is  saved  that  contains  a  one-to-one  mapping  of  every  laser  LyJ.Uj.UJ.U.U  !U 

instance  location  to  its  corresponding  screen  image  location.  This 

produces  a  much  larger  file  but  reduces  the  number  and  type  of  Figure  6.  4-Point  Initialization 
runtime  calculations. 

3. 1.3. 3  Runtime 

There  are  two  runtime  methods.  Each  initialization  method  is  paired  with  its  appropriate  runtime 
application  and  can  only  read  one  type  of  initialization  output  file.  However,  both  runtime 
applications  perform  the  same  task  and  will  not  be  differentiated. 

A  mouse  resource  window  that  models  a  three-button  mouse  appears 
after  executing  the  runtime  program.  This  window  allows  the  user  to 
select  the  way  in  which  the  laser  pointer  should  be  used.  If  there  are 
no  buttons  selected,  the  system  mouse  cursor  only  follows  the 
operator’s  laser  dot.  If  the  first  button  is  selected  (Fig.  7),  the  runtime 
software  will  generate  a  left  mouse  button  down  event  the  first  time 
the  laser  dot  is  seen.  A  mouse  button  up  event  will  be  generated  when 
the  laser  is  turned  off.  The  same  holds  true  for  the  other  two  buttons 
and  their  respective  operations.  This  allows  operators  to  wirelessly 
manipulate  a  GUI. 


Figure  7. 

Mouse  Resource  Window 


3.2  Continuous  Speech  Recognition 


An  essential  component  to  keyboardless  interaction  is  the  use  of  speech  recognition.  Using  a 
COTS  continuous  speech  recognition  system  called  HARK  from  BBN  Systems  and  Technologies, 
operators  are  able  to  interact  with  applications  using  a  wireless  microphone.  In  order  to  voice 
enable  an  application,  a  grammar  set  is  defined  to  trigger  the  actions  desired.  A  very  important 
capability  of  the  HARK  system  is  that  it  is  speaker  independent.  Any  user  can  immediately 
interact  via  voice  without  having  to  train  the  system  for  his/her  voice. 

Users  are  able  to  manipulate  windows,  the  background  and  alter  the  mode  of  the  laser  pointer.  By 
saying  the  command,  “Minimize  Window”  the  window  manager  will  cause  the  active  window  to 
be  minimized  just  as  if  it  had  been  performed  through  the  system  keyboard.  If  the  user  utters, 
“Enter  Freehand  Mode”,  the  application  will  cause  the  system  mouse  to  emulate  a  grease  pencil. 
When  the  user  selects  the  leftmost  button  of  the  mouse  resource  window,  he  can  make  freehand 
annotations  to  the  display.  Simple  geometric  shapes  can  be  drawn  including  circles,  boxes,  and 
lines  using  speech  commands  and  defining  point  locations  with  the  laser  pointer.  Annotation 
colors  and  line  characteristics  can  also  be  changed  through  speech  commands  (Fig.  8). 


Figure  8.  Electronic  Grease  Pencil 

The  speech  recognition  program  is  a  distributed  application.  The  recognition  occurs  on  a  lower 
end  machine  and  through  multicast,  broadcasts  a  code  representing  the  understood  phrases.  There 
are  two  primary  reasons  for  using  multicast  packets.  First,  connected  sockets  need  to  be  started  in 


order,  while  multicast  sockets  do  not.  Second,  multicast  packets  are  broadcast  and  all  applications 
that  want  to  support  speech  recognition  can  listen  to  the  understood  phrase  codes. 

4.  DataWall  Implementations 

AFRL/IF  has  successfully  implemented  a  number  of  Interactive  DataWalls  each  consisting  of 
three  horizontally  tiled  video  projectors.  The  first  implementation,  which  serves  as  a  development 
system,  uses  CRT  projectors  each  displaying  1600  x  1200  pixels  for  a  total  display  resolution  of 
4800  x  1200  pixels  across  a  screen  area  12’  x  3’  (Fig.  lb).  This  far  exceeds  the  state-of-the-art  in 
single  element  display  systems.  A  Silicon  Graphics,  Inc.  (SGI)  Onyx  workstation  with  three 
Reality  Engines  drives  the  display.  In  addition,  each  projector  has  a  video  bandwidth  capacity 
approaching  2500  x  2000  pixels,  which  could  yield  a  future  DataWall  resolution  of  15  million 
pixels. 

In  the  interest  of  developing  a  less  costly  alternative  to  the  SGI-based  DataWall,  a  Sun 
workstation  based  version  was  successfully  implemented  to  support  the  Joint  Force  Air 
Component  Commander  (JFACC)  program.  Again  three  CRT  projectors  were  utilized  but  at  a 
slightly  reduced  resolution.  Each  projector  displays  1280  x  1024  pixels  for  a  total  display 
resolution  of  3840  x  1024  pixels  across  a  screen  area  12’  x  3’.  One  dual  processor  Sun  Ultra 
SPARC  60  workstation  and  two  Ultra  SPARC  30  workstations  drive  the  display.  It  also  has  the 
same  wireless  interaction  capabilities  as  the  SGI-based  DataWall. 

To  provide  a  deployable  version  of  the  Interactive  DataWall  to  support  the  testbed  for  the 
forward  deployable  element  of  the  Configurable  Aerospace  Command  and  Control  (CACC) 
Integrated  Technology  Thrust  Program  (ITTP),  a  Sun  workstation  based  Deployable  Interactive 
DataWall  (DID)  was  implemented.  One  dual  processor  Sun  Ultra  SPARC  60  and  two  Ultra 
SPARC  10s  drive  this  display.  It  is  housed  in  an  extensively  modified  Air  Force  S-530  A/G 
Standard  Rigid  Walled  shelter,  with  its  own  Tactical  Generator  Set  and  Environmental  Control 
Unit.  Due  to  the  unique,  short-throw,  rear-projection  requirements,  three  LCD  projectors  with 
special  short-throw  lenses  are  used.  In  this  configuration,  each  projector  displays  1024  x  768 
pixels  for  a  total  display  resolution  of  3072  x  768  across  a  screen  area  110”  x  27”  (Fig.  9).  It 
should  be  noted  that  even  in  this  most  reduced  resolution  configuration,  the  total  display 
resolution  still  exceeds  the  state-of-the-art  in  single  element  display  systems.  A  new  feature 
implemented  in  the  DID  is  window  encapsulated  live  NTSC  video  feeds.  This  allows  the 
commander  and  staff  to  view  and  manipulate  information  from  such  sources  as  real-time 
surveillance  video,  satellite  broadcasts,  and  VCR  mission  playbacks. 


Figure  9a.  The  Deployable  Interactive  DataWall  (DID) 


Figure  9b.  The  DID  Interior  Components 


5.  Technology  Transition 

A  joint  effort  of  AFRL/LF’s  ADII,  CACC,  and  Logistics  Support  In-house  groups,  resulted  in  the 
first  field  evaluation  and  trial  of  the  DID,  with  the  US  Army’s  10th  Mountain  Division  at  Ft. 
Drum,  New  York  in  its  Mountain  Peak  98-01  field  training  exercise.  The  S-530  A/G  expandable 
rigid  wall  shelter  containing  the  DID,  along  with  the  supporting  generators  and  air  handling 
systems,  were  transported  to  Ft.  Drum  on  20  August  1998  to  begin  the  field  trial.  The  DID  was 
set  up  at  the  Division  Main  headquarters  in  the  field  and  provided  the  Division  Commander  and 


his  battle  staff  with  the  capability  to  display,  interact  with,  and  manipulate  real-time  multi-source 
information  essential  to  the  successful  control  of  the  battle.  During  this  field  exercise,  personnel 
from  AFRL/IFS’s  Division  along  with  the  Commander  and  soldiers  from  the  10th  Mountain 
Division  operated  the  interactive  DataWall.  It  provided  the  following  capabilities: 

•  Near  real  time  sensor  surveillance  display/manipulation 

•  A  unified  interface  with  Army  Command  and  Control  software  (All  Source  Analysis 
System  (ASAS)  and  Remote  Work  Station  (RWS)) 

•  Ability  to  display  large  high-resolution  still  and  motion  Intelligence,  Surveillance  and 
Reconnaissance  (ISR)  photography  (Navy  P3  Orion  live  fly  video  surveillance  and  high 
resolution  digital  camera) 

•  A  new  medium  for  fusion  of  satellite  information  broadcasts  and  Command,  Control, 
Communications,  Computers,  Intelligence,  Surveillance  &  Reconnaissance  (C4ISR) 
systems  (Digital  Satellite  System  (DSS)  broadcast  television  for  weather  reports  and  CNN 
updates  in  conjunction  with  Air  Force  weather  radar  data) 

•  Ability  to  create  and  interactively  deliver  web  based  Microsoft  PowerPoint  briefings 

The  exercise  was  conducted  from  24  August  1998  to  2  September  1998.  The  Army  10th  Mountain 
Division  commanders  and  staff  users  heartily  endorsed  the  DID,  giving  it  an  A++  grade.  Since  this 
exercise,  multiple  service  representatives  have  requested  prototype  variations  for  support  of 
C4ISR  operations  both  CONUS  and  abroad. 

AFRL/IF  is  also  providing  expertise  to  the  C2  Battle  Lab  at  Hurlburt  Field,  and  to  AFRL/HE  at 
Wright-Patterson  Air  Force  Base  to  install  DataWalls  at  their  sites.  The  installation  at  AFRL/HE 
is  part  of  a  Memorandum  of  Agreement  (MOA)  that  defines  a  cooperative  relationship  between 
the  IF  and  HE  directorates  to  provide  their  respective  scientific  and  engineering  expertise  to 
further  the  development  and  application  of  technology  in  areas  of  mutual  interest. 

6.  Conclusions  and  Future  Work 

The  custom  laser  pointer  tracking  hardware  has  proven  extremely  effective  and  reliable.  Several 
copies  of  the  device  are  planned  for  production,  to  replace  the  PC/capture  card  systems  still  being 
used  in  Sun  DataWall  configurations.  In  addition  to  the  advantages  discussed  earlier,  the  devices 
also  reduce  space  and  power  requirements,  which  is  particularly  beneficial  in  the  DID 
configuration. 

Also  in  development  is  a  PC-based  Interactive  DataWall.  Microsoft’s  Windows  98  and  Windows 
NT  5.0  will  support  multiple  displays.  Therefore,  a  single  PC  with  multiple  graphics  cards  (one 
graphics  card  per  display  tile)  could  drive  a  tiled  display  system.  The  obvious  advantage  is  further 
reduced  hardware  and  software  costs  compared  to  Unix-based  workstations.  Preliminary  testing  is 
being  conducted  using  Microsoft  Windows  NT  4.0  with  special  drivers  for  the  multiple  graphics 
cards  in  lieu  of  Windows  NT  5.0,  which  is  yet  to  be  released. 

Also  planned  is  the  development  of  a  Transportable  Interactive  DataWall  that  can  be 
disassembled,  transported,  and  reassembled  in  a  short  period  of  time.  Unlike  the  Deployable 


Interactive  DataWall,  this  version  is  intended  for  indoor  use.  It  will  most  likely  be  PC-based  and 
will  be  designed  to  fit  through  a  conventional  3’  doorway. 

Other  novel  HCI  devices  are  also  being  researched.  One  in  particular  is  using  a  COTS  magnetic 
position  tracking  system.  Although  these  systems  are  designed  for  3D  interaction,  and  the 
Interactive  DataWall  is  a  2D  environment,  the  appeal  of  attempting  to  integrate  this  type  of  input 
device  is  the  potential  to  further  simplify  the  configuration.  It  will  eliminate  the  video  cameras,  but 
more  importantly  is  able  to  track  multiple  receivers.  This  in  turn  may  allow  the  capability  of 
multiple  simultaneous  pointers. 

The  ability  to  support  multiple  simultaneous  users  is  the  Interactive  DataWall’ s  greatest 
limitation.  Although  multiple  users  can  interact  with  the  display  through  speech  and  laser  pointer, 
they  cannot  simultaneously.  The  problem  with  the  current  laser  pointer- tracking  paradigm  is  the 
inability  to  differentiate  multiple  red  laser  pointers  on  the  screen.  Research  has  been  conducted  by 
the  ADII  team  using  a  recently  COTS  available  green  laser  pointer.  While  preliminary  results 
revealed  the  ability  to  differentiate  the  red  and  green  lasers,  it  would  add  significant  complexity  to 
the  configuration,  requiring  separate  video  cameras  and  pointer  tracking  hardware  per  display  tile, 
to  make  it  a  two  person  system.  Each  additional  simultaneous  user  increases  the  complexity 
linearly.  Green  laser  technology  is  much  more  expensive  to  scale  it  down  to  the  size  of  a  pen, 
compared  to  the  widely  available  red  laser  pens.  Also,  there  are  no  other  colors  currently  available 
in  this  size  laser.  Yet  another  reason  the  magnetic  position  tracking  may  be  a  more  viable  option. 

If  the  multiple  pointer-tracking  problem  is  solved,  there  is  still  the  issue  of  the  window  manager 
supporting  multiple  cursors.  Currently  only  one  cursor  is  supported  under  the  X-META-X 
window  management  method.  To  assure  all  legacy  applications  can  be  supported  on  the 
Interactive  DataWall  without  modification,  has  quelled  our  desire  to  develop  a  custom  solution  to 
the  window  manager.  Hopefully  the  increasing  popularity  of  tiled  displays  will  motivate  the 
commercial  sector  to  develop  an  off-the-shelf  solution. 

Multiple  speaker  recognition  is  also  desired.  Currently,  a  copy  of  the  speech  recognition 
application  has  to  be  running  for  each  user  in  a  multi-user  environment.  Ideally  a  single  speech 
recognition  suite  should  accommodate  multiple  users.  Also,  once  the  multiple  pointer  issue  is 
solved,  the  problem  of  associating  a  user’s  voice  with  a  particular  pointer  must  still  be  resolved. 

The  Interactive  DataWall  has  been  demonstrated  to  a  number  of  visitors  at  AFRL/IF  at  Rome 
Research  Site.  It  has  been  extremely  well  received,  so  the  ADII  team  is  confident  that  our 
research  efforts  are  headed  in  the  right  direction  for  next  generation  Command  and  Control 
systems.  Although  a  considerable  amount  of  research  and  development  remains,  an  extremely 
capable  system  has  been  created,  integrating  a  significant  amount  of  commercially  available 
hardware  and  software. 
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